Synthetic image formation signal processing hardware for vignetted optoelectronic arrays, lensless cameras, and integrated camera-displays

ABSTRACT

Hardware arrangement performing electronic image formation and refinement from overlapping measurement vignettes captured by an array of image sensors and associated micro-optics are presented. The invention is directed to a new type of image formation system that combines readily-fabricated micro-optical structures, a two-dimensional image sensor array with electronic or digital image processing to actually construct the image. Image formation is performed without a conventional large shared lens and associated separation distance between lens and image sensor, resulting in a lensless camera. In an application, a readily fabricatable LED array is used as a light-field sensor. In an application, the LED array further serves as a color lensless camera. In an application, the LED array also serves as an image display. In an application, the LED array further serves as a color image display. In an embodiment, one or more synergistic features of an integrated camera/display surface are realized.

CROSS-REFERENCE TO RELATED APPLICATION

This regular U.S. patent application is a continuation of U.S.application Ser. No. 12/471,275 filed May 22, 2009 now U.S. Pat. No.8,125,559 B2; which claims the benefit of priority under 35 U.S.C. 119from provisional U.S. patent application No. 61/128,968, filed on May25, 2008. Application Ser. Nos. 12/471,275 and 61/128,968 are fullyincorporated herein by reference for all purposes.

FIELD OF THE INVENTION

This invention pertains to electronic image formation and refinementfrom overlapping measurement vignettes captured by an array of imagesensors and associated micro-optics. The present invention is directedin major part to an alternate image formation system that combinesreadily-fabricated micro-optical structures, a two-dimensional imagesensor array with electronic or digital image processing to actuallyconstruct the image.

BACKGROUND OF HE INVENTION

Image formation as used in camera obscura and photography dates back tothe pinhole camera discovery, experiments, descriptions, anddevelopments of Chinese philosopher Mozi (470 BC to 390 BC), Greekphilosopher Aristotle (384 to 322 BC), and Iraqi scientist Ibnal-Haitham (965-1039 AD). As depicted in FIG. 1 a, reflected or emittedlight from objects or light sources in a scene radiate in alldirections, but a small pinhole aperture limits the rays of light tothose collinear with both the pinhole aperture and the object or sourcein the scene. Light rays in other directions are blocked and thusprevented from scattering into the aforementioned collinear rays. Thecollinear rays then project undiffused and without comingling into theregion on the other side of the pinhole aperture where at any distancethey can be projected onto a surface, albeit with various distortions(due to irregularities in the aperture shape, degree of curvature of theprojection surface, etc.) and at low light intensity (since most of thelight rays are by necessity prevented from traveling through theaperture. Moving the projection surface closer to the pinhole aperturemakes the projected image smaller and brighter, while moving theprojection surface farther from pinhole aperture makes the projectedimage larger and dimmer. The image forms naturally at any distance byvirtue of the organization of the light rays passed through the pinholeaperture. In recent years CCD cameras using pinhole apertures ratherthan lenses have become commercially available.

In more modern mainstream forms of optical systems, a lens or system oflenses is employed for image formation for light passing through anaperture. A simple representative example is depicted in FIG. 1 b,wherein objects or light sources in a scene radiate light in alldirections and a significant angular sector of this is captured by alens of material, dimensions, and curvatures to systematically bend thedirection of travel of the captured angular sector of radiated light. Inparticular, the lens material, dimensions, and curvatures are such thatall light rays emanating and spreading from a given location on anobject or light source within a scene that are captured by the lens arebent so as to converge at a single point on the opposite side of thelens. In particular, the lens material, dimensions, and curvatures arecharacterized by a constant f (called the focal length), and if thedistance between the lens and an object or light source within a sceneis of the value A, the image forms in focus at a point at distance B onthe opposite side of the lens where 1/A+1/B=1/f (this relation known asthe “Lens Law”). At distances on the opposite side of the lens that areshorter than the distance B, the light rays have not yet come closeenough to converge, causing an out of focus image. At distances on theopposite side of the lens greater than the distance B, the light rayshave crossed each other and are spreading apart, also causing an out offocus image.

Both these approaches require a significant separation distance betweenthe (lens or pinhole) aperture and the image formation surface. Physicallimitations of these and other aspects of camera technology thus createa required appreciable depth or thickness of the optical imaging system.They also require all light for the resulting image to pass through anaperture.

The linear image sensors employed in fax machines, PDF™ scanners, andmany types of digital photocopiers use direct and localized-transfer“contact” imaging arrangements. A geometrically-linear array of lightsensors are positioned effectively directly upon a flat object such as apaper document that illuminated from the sides or other in other ways.Significant angular sectors of light radiating from a given spot on theflat object are captured by an extremely local light sensor element inthe linear array of light sensors. An example of this is shown in FIG. 1c. In many embodiments, each light sensor individually captures thelight value of a single pixel of the final generated image. In most faxmachines, PDF™ scanners, and many types of digital photocopiers, thegeometrically-linear image sensor array is either moved over the surfaceof the flat object (as in a scanner) or the flat object is moved overthe geometrically-linear image sensor array (as in a fax machine). Theseare suggested in the depiction of FIG. 1 d.

This differs from the lens and pinhole imaging systems (which require acentral aperture and free propagation space as the image formationmechanism) represented by FIGS. 1 a-1 b and the contact imaging systems(which require close to direct contact with the object an arrangementcoexistent with illumination) represented by FIGS. 1 d-1 e. In thissystem a spatial array of light sensors (typically relatively large, forexample with height and width dimensions of a few inches) performsmeasurements on the incoming light field, and imaging is obtainednumerically.

SUMMARY OF THE INVENTION

The present patent is directed to numerically-obtained image formationfrom data obtained by light sensor arrays (such as the rectangular arraydepicted in FIG. 1 via signal processing operations. The resultingsystem and method do not require a large bulky imaging lens nor thephysical separation distances required by imaging lenses or pinholecamera arrangements. The sensor for the resulting system can be flat orcurved, or may be integrated into LCD and LED display screens so as toobtain more direct eye contact and more natural telepresenceexperiences.

In an exemplary embodiment, it is assumed the image sensor array isprovided with a micro-optic structure, such as an array of micro-lensesor a suitably perforated optical mask, which provides a significantoccultation of the observed scene to each sensor so that each vignettecaptures a slightly different portion of the observed scene.

In an embodiment provided for by the invention, each LED in an LED arrayis, by various manners as may be advantageous, sequentially selected tooperate in light detecting mode while others adjacent to it are operatedin a light emitting mode or an idle mode.

In an exemplary embodiment, the image sensor array need not be planar,for example it may be rendered as a curved surface.

In an exemplary embodiment, image formation is realized by performinglinear mathematical transformations on the measurements made by thesensor array.

In an exemplary embodiment the type of image formed can be somewhatdifferent that that produced by a traditional photographic or videocamera by nature of the physical size and resultant spatial span of theassumed image sensor array being considerably larger than that of a lensor pinhole aperture.

In an exemplary embodiment, because the image formation is performed ina signal processing domain, it is possible to modify or modulate aspectsof the imaging as a function of a configuration command or controlsignal.

Additionally, the mathematics, imaging, and other aspects of theresultant technology may provide new understanding and appreciations forthe types and qualities of vision possible with the otherwisedeeply-studied single-lens (apposition) compound-eyes found inarthropods such as insects and crustaceans (these comprising many smallseparate “simple eye” elements comprising a cornea and/or lens andphotoreceptor). To date, the visual image-formation sensation oforganisms possessing single-lens apposition compound-eyes is thought tobe primitive and of limited resolution, providing perhaps more thefunction of a motion detector than that of the static image capabilitiesassociated with human vision. Should the possible static imagecapabilities of single-lens compound-eyes be revealed to be moresignificant than had been appreciated when signal processing imageformation is included in the assessment, various aspects of thesecreations of nature may be re-examined for use in new methods of imagingtechnology.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 a depicts the exemplary use of a small pinhole aperture employedfor image formation for light passing through an aperture as may be usedin a “pinhole camera” or camera obscura.

FIG. 1 b shows the exemplary use of a lens or system of lenses employedfor image formation for light passing through an aperture onto a focalplane on the opposite side of the lens.

FIG. 1 c depicts a “contact” imaging arrangement comprising ageometrically-linear array of light sensors positioned effectivelydirectly upon a flat illuminated object such as a sheet of paper.

FIG. 1 d depicts a geometrically-linear image sensor array that may bemoved over the surface of the flat object, as in a flatbed scanner, orthe flat object is moved over the geometrically-linear image sensorarray, as in a fax machine.

FIG. 1 e shows a side view of such arrangements, depicting light raysdirected to the geometrically-linear image sensor array.

FIG. 1 f depicts an exemplary direct view of a two-dimensionalmicro-optic structure and light sensor array as provided for by theinvention.

FIG. 2 a depicts an exemplary embodiment comprising a micro-opticstructure, a light sensor array, an image formation signal processingoperation and an optional additional subsequent image processingoperations.

FIG. 2 b depicts a variation of the embodiment of FIG. 2 a wherein themicro-optic structure and light sensor array are grouped into a firstsubsystem, and the image formation signal processing operation andsubsequent image processing operations are grouped into a secondsubsystem.

FIG. 2 c depicts a variation of the embodiment of FIG. 2 a wherein themicro-optic structure, light sensor array, and image formation signalprocessing operation are grouped into a common subsystem.

FIG. 2 d depicts a variation of the embodiment of FIG. 2 a wherein eachof the micro-optic structure, light sensor array, image formation signalprocessing operation, and subsequent image processing operations aregrouped into a common subsystem.

FIG. 3 a shows an exemplary arrangement wherein an individuallight-sensing element is effectively attached or proximate to acolumnated aperture element. The columnated aperture element may beimplemented as a punctuated mask, structured or etched deposition,micro-array of hollow columnated tubules, etc., and comprises a passageway for light to travel from the outside world to the individuallight-sensing element.

FIG. 3 b shows a variation on the arrangement of FIG. 3 a wherein theindividual light-sensing element is separated from the columnatedaperture element by a gap.

FIGS. 4 a and 4 b, respectively, depict exemplary variations on thearrangements of FIGS. 3 a and 3 b wherein the columnated apertureelement of FIGS. 3 a and 3 b has been replaced by a lens.

FIGS. 4 c and 4 d, respectively, depict other exemplary variations onthe arrangements of FIGS. 3 a and 3 b wherein the columnated apertureelement has been fitted with a lens on the exterior end.

FIGS. 4 e and 4 f, respectively, depict other exemplary variations onthe arrangements of FIGS. 3 a and 3 b wherein the columnated apertureelements has been fitted with a lens on the interior end.

FIGS. 4 g and 4 h, respectively, depict other exemplary variations onthe arrangements of FIGS. 4 a and 4 b wherein the lens is depicted asconcave-planar.

FIGS. 5 a and 5 b illustrate a representative example of lightreflecting or light emitting image elements and an exemplary array oflight sensing elements with associated columnated aperture elements.Exemplary rays of light are either passed (FIG. 5 a) or blocked (FIG. 5b) by the columnated aperture elements

FIG. 6 depicts a scene image of a glass of water that will be thesubject image for FIGS. 7 a and 7 b.

FIG. 7 a depicts an exemplary collection of circular or ellipticalvignettes, each created by a particular micro optic structure.

FIG. 7 b depicts a variation on FIG. 7 a with more or square-shaped orrectangular-shaped vignettes, each created by a particular micro opticstructure.

FIG. 8 illustrates the process by which the degree of vignette overlapincreases as separation between the object in the scene and its distancefrom the micro-optic structure and light sensor array increases.

FIGS. 9 a-9 d illustrate how the degree of vignette overlap increasesfrom 0% to values approaching 100% as the separation distance between ascene object and the micro-optic structure and light sensor arrayincreases.

FIG. 10 a depicts a scene object that slanted with respect to the camerasensor that will result in a different capture range than the morevertical scene object, and those at the top of the depicted slantedscene object surface will have more overlapping vignettes than those atthe bottom of the depicted slanted scene object surface.

FIGS. 10 b and 10 c depict scene objects with curved surfaces that willresult in varying degrees of vignette overlapping over the scene objectsurface.

FIG. 11 a depicts an exemplary three-dimensional scene involving twoblocks and a spherical ball as observed by a two-dimensional micro-opticstructure and light sensor array.

FIG. 11 b shows a view akin to that observed by the two-dimensionalmicro-optic structure and light sensor array, with the dotted area shownin FIG. 11 c in an expanded view.

FIG. 12 a shows an exemplary columnated aperture covered by a singlepixel light sensor. FIG. 12 b shows a variation of FIG. 12 b wherein thesingle pixel light sensor is smaller than the area of the columnatedaperture.

FIG. 12 c shows an exemplary realization of a multiple-pixel lightsensing element.

FIG. 13 illustrates a procedure for obtaining a linear transformationand a computed inverse transformation that are matched to a particulartwo-dimensional micro-optic structure and light sensor array.

FIG. 14 shows how the resulting computed inverse transformationassociated with the measured given two-dimensional micro-optic structureand light sensor array can then be used for image formation.

FIG. 15 depicts the aperturing of a pixilated test image such as mayserve as the test image in FIG. 13.

FIG. 16 shows various displacements of an A×A aperture within an N×Nimage pixel array.

FIG. 17 depicts how a border of additional pixels in the light sensorarray can be adapted to provide data for the overreach pixels.

FIG. 18 a-18 d depict various circuits demonstrating approaches todetecting light with an LED.

FIG. 19 shows a simple illustrative variation of the circuit of FIG. 18a where a SPDT switch is introduced for controlling the function of LED1 at any given time.

FIG. 20 illustrates how two LEDscan share the amplifier circuit and anemitting LED while in alternating roles of emitting and detecting light.

FIG. 21 shows an illustrative variation of the arrangement of FIG. 20when a second switch is used to determine which LED is illuminatedresponsive to the light levels detected by LED1 or LED2.

FIG. 22 depicts a further illustrative variation of FIG. 21 wherein LED1and LED2 can reproduce a light distribution measured by LED 101 and LED102 while LED101 and LED2 can reproduce a light distribution measured byLED1 and LED2, all of this humanly perceived as occurringsimultaneously.

FIG. 23 depicts an illustrative circuit arrangement wherein an LEDconnected to a detection buffer/amplifier (as in FIG. 18 a) is alsoconnected to a binary-signal-controlled analog switch element.

FIGS. 24 a-24 c depict exemplary state diagrams for the operation of theLED and the use of input signals and output signals. FIG. 24 d depictsan exemplary state transition diagram reflecting the considerations ofFIGS. 24 a-24 c.

FIG. 25 illustrates an exemplary analog circuit method of generating apulse-width modulated waveform.

FIG. 26 a illustrates a mode control signal possessing a non-symmetricduty-cycle that favors emission time duration and limits detection timeduration.

FIG. 26 b depicts a simplified representation of FIG. 23 as is useful ininterpreting FIG. 26 a.

FIG. 27 a depicts the exemplary use of a sample/hold arrangement. FIG.27 b depicts exemplary relative timing for the detection signal and thesampling control signal.

FIG. 28 shows the combination of the arrangement of FIG. 27 a with thearrangement of FIG. 26 b.

FIG. 29 depicts an exemplary analog demultiplexing function.

FIG. 30 depicts an exemplary arrangement to sequentially select from aplurality of detection amplifier outputs.

FIG. 31 depicts a combination of the arrangements of FIGS. 29 and 30.

FIG. 32 a depicts a selectable grounding capability for atwo-dimensional array of LEDs. FIG. 32 b depicts an adaptation of FIG.32 a controlled by an address decoder so that the selected subset can beassociated with a unique binary address.

FIG. 33 depicts an exemplary combination of the arrangements of FIG. 28,FIG. 30, and FIG. 32 b to form a highly scalable LED array display thatalso functions as a light field detector.

FIGS. 34 a and 34 b depict exemplary functional cells that may be usedin a large scale array.

FIGS. 35 a-35 c depict adaptations of the digital circuit measurementand display arrangement of FIG. 18 d into the construction developedthus far.

FIGS. 36 a-36 b depict exemplary embodiments of an overall systemprovided for by the invention.

FIG. 37 a and FIG. 37 b depict a view of the resulting synergisticcombinations of methods and elements as provided for by the invention.

FIG. 38 a illustrated the human eye sensitivity to the spectrum ofwavelengths of visible light.

FIG. 38 b and 38 c depict exemplary positionings of three approximately50 nm basspands in portions of the visible spectrum near peaks of thecone responses and away from areas of significant spectral overlap.

FIG. 39 a depicts a mimic of the cone responses of the human eyedepicted in FIG. 38 a. FIG. 39 b depicts an approximating the curves ofFIG. 38 b.

FIGS. 40 a-40 b depict a system employing narrowband detection spectralLED properties imposed by desirable emission characteristicssupplemented by detection spectral of additional LEDs providingadditional coverage bands.

FIGS. 41 a-41 b depict various “gamma correction” operation arrangementsfor application to measurement data streams provided by detection signalmultiplexing arrangements.

FIG. 42 depicts a metric space of device realizations for optoelectronicdevices.

FIG. 43 a-43 c depict various lattice arrangements for individual LEDs,stacks of LEDs, or groups of LEDs and/or electronics elements asprovided for by the invention.

FIG. 44 depicts a numerical micro-aperture aperature arrangement forsynthetic image formation of scene elements at different depths offield.

DETAILED DESCRIPTION

Referring to FIG. 2 a, incoming light 200 from a visual scene isoptically organized, filtered, etc. by at least one micro-opticstructure 201 to provide light field 201 a suitably structured forincoming presentation to a light sensor array 202. The light sensorarray 202 produces a representation (electrical, digital, numeric, etc.)202 a of the incoming light field 201 a suitable for presentation to asignal processing operation 203 configured to provide image formation.In an example embodiment, the representation 202 a of the incoming lightfield is not a visually useful electronic or digital image as itcomprises pixels having amplitude values resulting from summations oflight levels across entire individual image vignettes. In an exampleembodiment, the image formation signal processing operation 203comprises a linear transformation whose character depends on the opticalorganization the micro-optic structure 201 imposes on the incoming light200 from the visual scene. The image formation signal processingoperation 203 produces a visually useful electronic or digital image 203a, which may be used directly or which may be provided to subsequentimage processing operations 204 to result in an outgoing digital image205.

An embodiment may provide for the image formation signal processingoperation 203 to be adjustable responsive to provided configurationcommands or control signals 207. An embodiment may provide for the imageprocessing operations 204 could be configurable and adjustable andinclude the option for a no-change pass-through (no-operation)operation. The image processing operations 204 may be adjustableresponsive to provided configuration commands or control signals 208.

In one adaptation of the arrangement of FIG. 2 a, the micro-opticstructure 201 and the light sensor array 202 are grouped into a firstsubsystem 211, and the image formation signal processing operation 203and the subsequent image processing operations 204 are grouped into asecond subsystem 212. This adaptation is depicted in FIG. 2 b. Theadaptation may provide for the image processing operations 204 to beconfigurable and adjustable, and include the option for a no-changepass-through (no-operation) operation. The image processing operations204 may be configured or adjusted responsive to provided configurationcommands or control signals 208.

In another adaptation of the arrangement of FIG. 2 a, the micro-opticstructure 201, the light sensor array 202, the image formation signalprocessing operation 203 are grouped into a subsystem 220. Anysubsequent image processing operations 204, should they be needed orprovided, are performed outside the subsystem 220. This adaptation isdepicted in FIG. 2 c.

In another adaptation of the arrangement of FIG. 2 a, the micro-opticstructure 201, the light sensor array 202, the image formation signalprocessing operation 203, and any provide support for subsequent imageprocessing operations 204 are grouped into a common subsystem 230. Thisadaptation is depicted in FIG. 2 d. In one arrangement of the adaptationof FIG. 2 d the image processing operations 204 could be configurableadjustable, and include the option for a no-change pass-through(no-operation) operation. The image processing operations 204 may beconfigured or adjusted responsive to provided configuration commands orcontrol signals 208.

Attention is now directed to the relationship and arrangementspertaining to the micro-optic structure 201 and the light sensor array202. FIG. 3 a shows an exemplary arrangement wherein an individuallight-sensing element 301 is effectively attached or proximate to acolumnated aperture element 302. The columnated aperture element 302 maybe implemented as a punctuated mask, structured or etched deposition,micro-array of hollow columnated tubules, etc., and comprises a passageway for light to travel from the outside world to the individuallight-sensing element 301 but is otherwise opaque.

The depth of the columnated aperture element 302 is such that lightapproaching the arrangement from certain angles, such as those depictedas light rays 321 and 322, can reach the light-sensing element 301 butlight approaching the arrangement from all other angles, such as thelight ray 323, are blocked and absorbed by the walls of the columnatedaperture element 302. Light absorption by the walls of the columnatedaperture element 302 may be realized by the material used to create thecolumnated aperture element 302, by a depositional coating applied tothe walls of the columnated aperture element 302, or by other means. Theindividual instances of columnated aperture elements 302 within themicro-optic structure 201 and the individual light-sensing element 301within the light sensor array 202 is such that light from neighboringcolumnated aperture elements 302 and individual light-sensing elements301 does not conduct or refract in such a ways as to create cross-talkor noise conditions among the individual light-sensing elements 301within the light sensor array 202.

FIG. 3 b shows a variation on the arrangement of FIG. 3 a wherein theindividual light-sensing element 301 is separated from the columnatedaperture element 302, by a gap 303. The arrangement involving the gap303 would need to be implemented such that that light from neighboringcolumnated aperture elements 302 and individual light-sensing elements301 does not conduct or refract in such a ways as to create cross-talkor noise conditions among the individual light-sensing elements 301within the light sensor array 202.

FIGS. 4 a and 4 b, respectively, depict exemplary variations on thearrangements of FIGS. 3 a and 3 b wherein the columnated apertureelement 302 of FIGS. 3 a and 3 b has been replaced by a lens 404. Herethe lens 404 is depicted as being convex-planar, but it is understoodthe lens 404 could be concave-planar, binary, Fresnel, etc., and in thecase of FIG. 4 b may also be convex, binary, Fresnel, etc. on the sideof the lens closest to the individual light-sensing elements 401.

FIGS. 4 c and 4 d, respectively, depict other exemplary variations onthe arrangements of FIGS. 3 a and 3 b wherein the columnated apertureelement 402 has been fitted with a lens 404 on the exterior end. Herethe lens 404 is depicted as being convex-planar, but it is understoodthe lens 404 could be concave-planar, binary, Fresnel, etc, and may alsobe convex, binary, Fresnel, etc. on the side of the lens closest to theindividual light-sensing elements 401.

FIGS. 4 e and 4 f, respectively, depict yet other exemplary variationson the arrangements of FIGS. 3 a and 3 b wherein the columnated apertureelement 402 has been fitted with a lens on the interior end. Here thelens 404 is depicted as being convex-planar, but it is understood thelens could be concave-planar, binary, Fresnel, etc., and in the case ofFIG. 4 f may also be convex, binary, Fresnel, etc. on the side of thelens closest to the individual light-sensing elements 401.

FIGS. 4 g and 4 h, respectively, depict still yet other exemplaryvariations on the arrangements of FIGS. 4 a and 4 b wherein the lens 404is depicted as concave-planar. In the case of FIG. 4 b may also beconvex, binary, Fresnel, etc. on the side of the lens closest to theindividual light-sensing elements 401.

Many other variations of these types and beyond may be chosen to createvarious situations that would generate usefully structured light 201 afrom the scene 201 in accordance with the principles of the invention aswould be clear to one skilled in the art and thus are provided for bythe invention.

As indicated earlier, the purpose of these micro-optic structure 201 andlight sensor array 202 arrangements generate usefully structured light201 a from the scene and in particular to partition the light from thescene so that each of the individual light-sensing elements 301, 401within the light sensor array 202 obtains a unique one of a plurality ofoverlapping vignettes derived from the incoming light from the scene.FIGS. 5 a and 5 b illustrate a representative example of lightreflecting or light emitting image elements 511-518 and an exemplaryarray of light sensing elements 501 a-501 c with associated columnatedaperture elements 502 a-502 c. Exemplary rays of light from the lightreflecting or light emitting image elements 511-518 are either passedthrough (FIG. 5 a) or blocked (FIG. 5 b) by the columnated apertureelements 502 a-502 c. It is noted that in this example, the exemplaryscene is planar and oriented perpendicularly to the array of lightemitting image elements 501 a-501 c.

To further appreciate the purpose of these micro-optic structure 201 andlight sensor array 202 arrangements, FIG. 6 depicts a scene image of aglass of water that will be the subject image for FIGS. 7 a and 7 b.FIG. 7 a depicts an exemplary collection of circular or ellipticalvignettes, each created by a particular micro optic structure(comprising, for example, elements such as 302, 402, 404, etc., asdescribed in conjunction with FIGS. 3 a-3 b and 4 a-4 g, a hole in amask, etc.) and observed by a particular light sensing elements 301, 401as described in conjunction with FIGS. 3 a-3 b and 4 a-4 g. FIG. 7 bdepicts a variation on FIG. 7 a with more or square-shaped orrectangular-shaped vignettes, each created by a particular micro opticstructure.

Note in comparison with the type of imaging done with pinholes andlenses (cf. FIGS. 1 a and 1 b), wherein the image is viewed from belowfor areas of the scene that are above the lens and viewed from above forareas of the image that are below the lens, these vignettes are viewedstraight-on. Under these circumstances the image produced is somewhatdifferent from that as would be formed by a lens or pinhole arrangementsuch as those depicted in FIGS. 1 a and 1 b. In some embodiments, it maybe advantageous to adjust the directional orientations of themicro-optic structures to various angles so as to obtain other times ofimage acquisition geometries.

Further, it is noted that the degree of overlap of the vignettes istypically a function of the separation distance between a scene objectand the micro-optic structure 201 and light sensor array 202. FIG. 8illustrates the process by which the degree of vignette overlapincreases as separation between the object in the scene and its distancefrom the micro-optic structure 201 and light sensor array 202 increases.In FIG. 8, light sensing element 801 is depicted with a conical lightcapture range bounded by boundaries 801 a and 801 b. When the object isin contact or near contact with the micro-optic structure 201 and lightsensor array 202, the size of each light capture area is very locallydefined, in this case depicted so that light reflected or radiating froman object location 811, 812, 813 immediately adjacent to a particularlight capture aperture and associated light sensing element 801, 802,803, etc. is capture only by that particular light capture aperture andassociated light sensing element. In FIG. 8, the spacing betweenneighboring object locations of 811, 812, 813 is uniform, and is thesame for neighboring object locations subsequently discussed objectlocations 821, 822, 823, 824 and 831, 832, 833, 834, 835, 836, 837.

At a farther separation distance, the light capture vignettes begin toboth expand and overlap. For example, at a farther separation distancethe conical light capture range for light sensing element 801 (boundedby boundaries 801 a and 801 b) expands to include additional objectlocations 821, 822, 823, 824. Further, light reflecting or radiatingfrom object locations 823, 824 are also captured by light sensingelement 802, meaning object locations 823, 824 are in an area of overlapof the conical light capture range for light sensing element 801 and theconical light capture range for light sensing element 802. At a yetfarther separation distance the conical light capture range for lightsensing element 801 (bounded by boundaries 801 a and 801 b) expands toinclude additional object locations 831, 832, 833, 834, 835, 836, 837.Of these, object locations 833, 834, 835, 836, 837 are also captured bylight sensing element 802 and 835, 836, 837 are also captured by lightsensing element 803. Thus object locations 835, 836, 837 are in an areaof three-way overlap of light sensing elements 801, 802, and 803. As canbe seen, a simple “similar triangles” argument from high-school geometryshows that the conical light capture range of a given light sensingelement expands linearly in width as the radial separation distanceincreases.

FIGS. 9 a-9 d illustrate how the degree of vignette overlap increasesfrom 0% to values approaching 100% as the separation distances between ascene object and the micro-optic structure and light sensor array. Whenthe scene object is in or very nearly in direct contact with themicro-optic structure 201 and light sensor array 202, the system can bedesigned so that there is no overlap between vignettes, as shown in FIG.9 a. This corresponds to the contact imaging systems such as depicted inFIGS. 1 d and 1 e. As the source of the image moves away from the sensorarray, the natural ray geometry brings more of the image scene into thelight capture aperture 302, 402 of each light sensing element 301, 402,801, etc. At some point in the increased separation, the portions of theimage scene will reflect or radiate light into more than one lightcapture aperture and its associated light sensing element, creating anoverlap in vignettes between at least adjacent light capture aperturesand associated light sensing elements as depicted in FIG. 9 b. As theseparation distance increases between scene object and the micro-opticstructure 201 and light sensor array 202, the overlap in vignettesbetween light capture apertures and associated light sensing elementsalso increases further, as suggested by FIG. 9 c. As the separationdistance between scene object and the micro-optic structure 201 andlight sensor array 202 becomes very large, the overlap in vignettesbetween light capture apertures and associated light sensing elementsbecome significant, and eventually to the point where they nearlycoincide, as suggested by FIG. 9 d.

This process and effect is not dissimilar from the decrease inresolution and resolving power of conventional lens-based opticalimaging systems as subject objects in the image appear increasingly farfrom the lens projecting a formed image on a resolution-limited sensingelement such as a CCD light sensing array film emulsion, or photographicplate. It also has implications for objects that are not coplanar to themicro-optic structure 201 and light sensor array 202. For example, inFIG. 10 a the slanted scene object surface has a different capture rangethan the more vertical scene object surface, and those at the top of thedepicted slanted scene object surface will have more overlappingvignettes than those at the bottom of the depicted slanted scene objectsurface. Similarly, scene objects with curved surfaces, such as thestrictly convex example of FIG. 10 a and the undulating curvatureexample of FIG. 10 b, will also experience varying degrees of vignetteoverlapping over the scene object surface.

As the object scene grows in geometry complexity, additional phenomenabecome apparent. FIG. 11 a depicts an exemplary three-dimensional sceneinvolving two blocks and a spherical ball as observed by atwo-dimensional micro-optic structure 201 and light sensor array 202,the latter shown in profile. The two-dimensional micro-optic structure201 and light sensor array 202 could, for example, appear in direct viewas shown in FIG. 1 f. FIG. 11 b shows a view akin to that observed bythe two-dimensional micro-optic structure 201 and light sensor array202, with the dotted area shown in FIG. 11 c in an expanded view. Theobject scene of FIG. 11 b involves a curved surface, with implicationsas discussed in conjunction with FIG. 10 b, edges of the cubes thatcomprise slanted surfaces, with implications as discussed in conjunctionwith FIG. 10 a, and boundary issues wherein the edge of a nearer objectabruptly ends and the surface of a more distant object appears. Theseall present image formation challenges to be considered and addressed.

Further, the various cases described thus far can comprise a variety oflight sensor elements. FIG. 12 a shows a simple case wherein the fullarea of an exemplary columnated aperture is covered by a single pixellight sensor. FIG. 12 b shows a variation of this wherein the singlepixel light sensor is smaller than the area of the columnated aperture.An alternate implementation of effect realized by the arrangement ofFIG. 12 b can be obtained by placing a restrictive mask over the lightsensor, so as to reduce its light-sensing area, in the implementation ofFIG. 12 a. Other alternative implementations are clear to one skilled inthe art and are provided for by the invention. Also, each light sensoritself can be single pixel, multi-pixel, monochrome, color,multiple-wavelength, etc. This is provided for by the invention. FIG. 12c shows an exemplary realization of a multiple-pixel light sensingelement; either or both stacked and/or adjacent color andmultiple-wavelength light sensors.

In one embodiment of the invention, image formation is realized byperforming linear transformations on the measurements made by the sensorarray. The type and coefficient particulars of the resulting action ofthe linear transformation depend on details of the two-dimensionalmicro-optic structure 201 and light sensor array 202, as well as thedetails of the type of imaging desired and ranges of separation distanceinvolved in the scene to be observed. In general, particularly for earlyproduct manufacture, the two-dimensional micro-optic structure 201 andlight sensor array 202 may be difficult to reproducibly fabricate withinnecessarily accurate tolerances so that linear transformations can bemade to match an a priori family of linear transformation coefficients.Further, it is likely desirable that a two-dimensional micro-opticstructure 201 and light sensor array 202 with spot variations or defects(either present at time of manufacture or appearing as the result ofaging or damage) or systemic parameter drift over time or temperaturemay also prevent high performance with the use of an a priori family oflinear transformation coefficients. FIG. 13 illustrates a procedure forobtaining a linear transformation that is matched to a particulartwo-dimensional micro-optic structure 201 and light sensor array 202.This procedure may be applied to each two-dimensional micro-opticstructure 201 and light sensor array 202 manufactured, or to atwo-dimensional micro-optic structure 201 and light sensor array 202arrangement representative of a group or lot within specific tolerances,or for an individual two-dimensional micro-optic structure 201 and lightsensor array 202 at one or more times over its lifetime. In oneembodiment, the arrangement of FIG. 13, or variations of it apparent toone skilled in the art, may be built into a camera product for thepurpose of ongoing or as-needed self-contained calibration.

In FIG. 13, a light field from a (for example, pixilated) test image1300 is directed to a given two-dimensional micro-optic structure 1301and light sensor array 1302 to produced measured data that may be storedin a record 1304 for use in an analysis function 1305. In anotherimplementation, the measured data from the light sensor array 1302 maybe passed directly to the analysis function 1305. The analysis function1305 produces a measured linear transformation 1306. From this, theinverse of the measured linear transformation is computed 1307 resultingin a computed inverse transformation 1308 associated with the measuredgiven two-dimensional micro-optic structure 1301 and light sensor array1302. The inverse transformation may be computed in any number of ways,including deconvolution, spectral inversion, generalizedinverse/pseudoinverse, Gaussian Elimination, etc. Deconvolution forimages is well-known; see for example see Peter A. Jansson,Deconvolution of Images and Spectra (updated Second Edition availablefrom Dover, N.Y., 2008, ISBN 9780486453255) and many contemporaryreferences on image processing. An exemplary method using a(Moore-Penrose) psuedoinverse is provided later in the specification.

It is noted that the computed inverse transformation 1308 may take on anumber of structural forms. In one embodiment, the computed inversetransformation 1308 is represented by a 4-dimensional array equivalentto a tensor operating on one or more matrices (equivalent to atwo-dimensional data array) so as to produce another one or morematrices (equivalent to a two-dimensional data array). In anotherembodiment, wherein image data is organized as a (long) vector or datastream, the computed inverse transformation 1308 is represented by a(large) matrix or equivalent arrangement for linear computation. It isnoted that in many implementations it may be advantageous to perform thecomputations involved with the computed inverse transformation 1308directly on a data stream produced by sequenced scans of the lightsensor array 1302.

FIG. 14 shows how the resulting computed inverse transformation 1308associated with the measured given two-dimensional micro-optic structure1301 and light sensor array 1302 can then be used for image formation. Ascene image 1400 is applied to associated two-dimensional micro-opticstructure 1301 and light sensor array 1302 to produce data that isoperated on by the computed inverse transformation 1308 associated withthe measured given two-dimensional micro-optic structure 1301 and lightsensor array 1302. This produces at least an image of at least a rawform 1401 which can either be used directly or subjected to additionalimage processing 1402 to produce enhanced images.

In order for the inverse transformation 1308 to exist mathematically forthe number of pixels comprised in a final image 1401, the measuredlinear transformation 1306 must have sufficient rank (in the sense oflinear transformations, matrices, and operators). This is equivalent tothe condition that in order to solve for N unknown variable there mustbe at least N linearly-independent equations. Should there be fewer thanN linearly-independent equations, there are not enough equations tosolve for the values of the pixels in the a final image 1401;accordingly the rank of the measured linear transformation 1306 is lessthan N. Should the number of equations be more than N, the system can beover-specified. This is not a problem should the extra equations belinearly-dependent on some subset of N linearly-independent equations.In actuality, however, this cannot be casually or even stringentlyassured in measurements made of a physical system. Thus the giventwo-dimensional micro-optic structure 1301 and light sensor array 1302,together with any actions imposed on the data by the interface to lightsensor array 1302, must be such that at least a sufficient number oflinearly independent equations be represented in the measured lineartransformation 1306, and additionally the computation 1307 resulting ina computed inverse transformation 1308 associated with the measuredgiven two-dimensional micro-optic structure 1301 and light sensor array1302 must be such that any over-specified data be used or sufficientlyparsed and discarded as appropriate.

FIG. 15 depicts exemplary aperturing of an exemplary pixilated testimage such as may serve as the test image 1301 in FIG. 13. The exemplaryapertures here are depicted as squares, but may take on other shapessuch as circles, ellipses, rectangles, diamonds, crosses, etc. as maycorrespond to the non-zero transmission patterns encountered in thedesign of the two-dimensional micro-optic structure 1301 and lightsensor array 1302. Aperture 1501 is comprises only pixels in the testimage pixel array and is fully populated with those pixels. Aperture1502 comprises fewer of these image pixels by virtue of a 1-pixeloverreach off the edge of the image in one direction. Aperture 1503comprises yet fewer of image pixels by virtue of a 2-pixel overreach offthe edge of the image in the opposite direction. Aperture 1504 comprisesyet fewer of image pixels by virtue of a 2-pixel overreach off the edgeof the image both orthogonal directions. As a result of the aperturesize, aperture 1504 results in the capture of only a single pixel fromthe test image.

Each light sensor thus measures a linear combination of the lightamplitudes of at least one pixel in the test image and typically as manypixels in the test image as the aperture permits. Thus the output fromeach light sensor effectively represents an equation, and in that eachlight sensor measures a distinct collection of pixels, there is astructure by which at least a number of linearly-independent equationscan be had. A key goal is to ensure that the arrangement results in anadequate number of linearly-independent equations so that an inverselinear transformation exists and can be computed. An additionalrequirement could include that the collection of equations are such thatthe resulting matrix associated with the resultant system of equationsis sufficiently non-singular to prevent numerical problems incalculating the computed inverse transformation 1308.

It can be readily seen that the number of measurements depends on thenumber of pixels in an image, the number of pixels in an aperture, andthe number of pixels the aperture is allowed to overreach the edges ofan image. Let the array be N×M, and consider the side of the image thathas dimension N. An aperture of size A on the corresponding side can bepositioned at N−(A−1) or N−A+1 locations along that side of the image,as can be confirmed by visual inspection of a diagram such as FIG. 16which shows various displacements of an A×A aperture within an N×N imagepixel array.

In the case of FIG. 15, N=7 and A=3 so N−A+1=5, which again can beconfirmed by visual inspection of FIG. 15.

Each permitted pixel of overreach adds a count of two to the quantityN−A+1, so for K pixels of permitted overreach, the number ofmeasurements is N−A−1+2K. If one pixel of overreach is permitted in FIG.15 (i.e, K=1), N−A−1+2K=7. If two pixels of overreach is permitted inFIG. 15 (i.e., K=2), N−A−1+2K=9.

This count scheme applies in each dimension so for an N×M test image thetotal number of measurements is (N−A+1+2K)*(M−A−1+2K). This count schemecan be adjusted for other types of aperture shapes and differingoverreach policies in each direction as is apparent to one skill in theart and as such are provided for by the invention.

In the case of a square N×N test image the count is the aforementionedformula evaluated with N=M, or (N−A+/+2K)². For the case of FIG. 15(N=7, A=3):

-   -   If no overreach is permitted (i.e., K=0), the test image        comprises 7×7=49 pixels while the total measurement count is        (7−3+1)²=25. In this case there are not enough equations to        recover the original 49 pixels, only enough for a 25-pixel        decimated version of the test image.    -   If a one pixel of overreach is permitted (i.e., K=1), the test        image comprises 7×7=49 pixels while the total measurement count        is (7−3+1+2)²=49. In this case there are exactly enough        equations to recover the original 49 pixels assuming all        measurements are of sufficient quality and that the system of        equations is not numerically unstable (due to a corresponding        near-singularity in the measured linear transformation 1306).    -   If a two pixel overreach is permitted (i.e., K=2), the test        image comprises 7×7=49 pixels while the total measurement count        is (7−3+1+4)²=81. In this case there are far more than enough        equations to recover the original 49 pixels should at least 49        of the equations be linearly independent.

For the system of equations to be over-specified, the apertures andoverreach must be engineered so that 2K+1>A. When the system ofequations is over-specified, some of the equations can be discarded, butthe extra equations can be used to build robustness into the computedinverse transformation 1308 as will be explained.

As to overreach processes, there are at least two possible approachesplus various combinations of these as provided for by the invention. Inone exemplary embodiment, the overreach pixel locations are ignored. Inanother exemplary approach, the recovered image 1401 is smaller than thelight sensor array 1302. The additional pixels in the light sensor array1302 provide data for the overreach pixels as depicted in FIG. 17.

The chosen aperture model and overreach policy is organized as assumingthe image to be calculated 1401 is set to be n-row by m-column. In oneembodiment, an over-specified the measured linear transformation 1308 isobtained for the N-row by M-column image.

Test images can be applied to obtain the measured linear transformation1308. Such a measurement can be obtained in any number of ways as wouldbe clear to one skilled in the art and these are provided for by theinvention. For example, individual pixels of an LCD or LED display canbe lit sequentially with calibrated brightnesses. In another example,sufficiently isolated combinations of pixels of an LCD or LED displaycan be lit sequentially with calibrated brightnesses. In anotherexample, a fixed test pattern with associated spatio-spectral propertiescan be used and more intensive computations used obtain the measuredlinear transformation 1308.

In one embodiment, the observed image is represented by a singleone-dimensional vector or data stream. This representation can beobtained in any number of ways as would be clear to one skilled in theart and these are provided for by the invention. For example, a pixel inan N-row by M-column image array, said pixel having row and column index(r, c), would be assigned to vector or data stream position M*r+c. Inanother example, the same pixel could be assigned to vector or datastream position r+N*c. In such an arrangement, the measured lineartransformation 1308 is a sparse N*M by n*m matrix, at least in abstractmathematical representation. For large values of N, n, M, and m, othercomputer representations of the measured linear transformation 1308,such forms of linked lists, may be used. If either of N>n and/or M>m,the resulting matrix is note square and the system of equations isover-specified.

The traditional solution to a linear vector equation comprising a lineartransformation by a matrix requires the matrix to be square andnonsingular. For a non-square matrix it is possible to use one form oranother of a generalized inverse or pseudo-inverse linear operator. Inan exemplary embodiment of interest, the Morse-Penrose pseudo-inverse isused. The Morse-Penrose pseudo-inverse employs a least-squares fittingof the inverse through the over-specified data in the originalrectangular matrix. This least-squares result builds a robustness intothe structure and performance of the inverse as outlier measurements arenaturally compensated for by the least-squares fitting.

As a simple demonstration, one skilled in the art can begin with a smallmonochrome pixel array, for example 20×20, comprising a recognizableimage with features that can be easily scrutinized. (Alternatively, atable of arbitrary numbers could be used in place of an image.) Theimage will be used as a scene image 1400. Its (square) array of pixelsis reorganized into a (400-entry) test image vector using one or theother of the N*r+c or r+N*c formulas described above. The aperturearrangement and measured linear transformation 1306 can be emulated by asmall aperture matrix, for example 2×2. For a simple illustrativeexample, all entries in the 2×2 aperture matrix can be set to 1. Anested loop can be used to perform conditional tests on iterated indicesto produce the emulated measured linear transformation 1306, which inthe case of 1-pixel overreach will be a large rectangular matrix of size400×441. A Morse-Penrose pseudo-inverse operation can be applied to thismatrix—for example if the calculations are done in the Mathematica®programming language, there is a primitive function available forperforming the Morse-Penrose pseudo-inverse—to produce the computedinverse transformation 1308 (in this case, a 441×400 matrix).

The image vector is operated on by the emulated measured lineartransformation 1306 (the 400×441 matrix) to produce a (441-entry)emulated light sensor array 1302 data output, representing an actualdata output vector or a data stream. If desired in this calculationsmall quantities of random noise or other deviations can be added to thenon-zero entries of the emulated measured linear transformation 1306(400×441) matrix to simulate imperfect measurements.

The Morse-Penrose pseudo-inverse computed inverse transformation 1308(441×400 matrix) is applied to the (441-entry) light sensor array datavector (data stream) to produce a (400-entry) vector representing thepixel values of the computed image 1401. This vector is organized into a20×20 array by inverting the previously chosen one of the N*r+c or r+N*cformulas. This 20×20 array is rendered as an image (or viewed as anarray of numbers) and compared to the original. In the case of no noiseto perturb the emulated measured linear transformation 1306, theoriginal image 1400 and calculated image B are identical. In the case ofa small amount of noise to perturb the emulated measured lineartransformation 1306, the Morse-Penrose pseudo-inverse performs more workand original image 1400 and calculated image 1401 differ somewhat.

Use of LED Arrays as Light Sensors and Time-Multiplexed Sensor-Displays

Light detection is typically performed by photosite CCD (charge-coupleddevice) elements, phototransistors, CMOS photodetectors, andphotodiodes. Photodiodes are often viewed as the simpliest and mostprimitive of these, and typically comprise a PIN(P-type/Intrinstic/N-type) junction rather than the more abrupt PIN(P-type/N-type) junction of conventional signal and rectifying diodes.

However, virtually all diodes are capable of photovoltaic properties tosome extent. In particular, LEDs, which are diodes that have beenstructured and doped specific types of optimized light emission, canalso behave as at least low-performance photodiodes. In popular circlesForrest M. Mims has often been credited as calling attention to the factthat that a conventional LED can be used as a photovoltaic lightdetector as well as a light emitter (Mims III, Forrest M. “SunPhotometer with Light-emitting diodes as spectrally selective detectors”Applied Optics. Vol. 31, No. 33. Nov. 20, 1992), and that as aphotodetector the LEDs can exhibit spectral selectivity similar to thatof the LED's emission wavelength. Additionally, LEDs also exhibit otherreadily measurable photo-responsive electrical properties, such asphotodiode-type photocurrents and related accumulations of charge in thejunction capacitance of the LED.

In an embodiment provided for by the invention, each LED in an array ofLEDs can be alternately used as a photodetector or as a light emitter.At any one time, each individual LED would be in one of three states:

-   -   A light emission state,    -   A light detection state,    -   An idle state.        as may be advantageous for various operating strategies. The        state transitions of each LED may be coordinated in a wide        variety of ways to afford various multiplexing, signal        distribution, and signal gathering schemes as may be        advantageous. The similarities between the spectral detection        band and the spectral emission bands of each of a plurality of        types of colored-light LED may be used to create a color        light-field sensor from a color LED array display such as that        currently employed in “LED TV” products and road-sign        color-image LED advertising signs. The various materials,        physical processes, structures, and fabrication techniques used        in creating the LED array and associated co-located electronics        (such as FETs, resistors, and capacitors) may be used to further        co-optimize a high performance monochrome LED array or color LED        array to work well as both an image display and light-field        sensor compatible with synthetic optics image formation        algorithms using methods, systems, and process such as those        aforedescribed.

Employing these constructions, the invention provides for an LED arrayimage display, used in place of a LCD image display, to serve as atime-multiplexed array of light emitter and light detector elements. Theresulting system does not require an interleaving or stacking offunctionally-differentiated (with respect to light detection and lightemission) elements. This is particularly advantageous as there is a vastsimplification in manufacturing and in fact close or precise alignmentwith current LED array image display manufacturing techniques andexisting LED array image display products.

FIG. 18 a through FIG. 24 depict circuits useful in demonstratingprinciples and signal management strategies that are employed in thisaspect of the invention. These initially introduce the concepts ofreceived light intensity measurement (“detection”) and varying lightemission intensity of an LED in terms of variations in D.C.(“direct-current”) voltages and currents. However, light intensitymeasurement (“detection”) may be accomplished by other means such as LEDcapacitance effects—for example reverse-biasing the LED to deposit aknown charge, removing the reverse bias, and then measuring the time forthe charge to then dissipate within the LED. Also, varying the lightemission intensity of an LED may be accomplished by other means such aspulse-width-modulation—for example, a duty-cycle of 50% yields 50% ofthe “constant-on” brightness, a duty-cycle of 50% yields 50% of the“constant-on” brightness, etc. These, too, are provided for by theinvention and will be considered again later as variations of theillustrative approaches provided below.

To begin, LED1 in FIG. 18 a is employed as a photodiode, generating avoltage with respect to ground responsive to the intensity of the lightreceived at the optically-exposed portion of the LED-structuredsemiconducting material. In particular, for at least a range of lightintensity levels the voltage generated by LED1 increases monotonicallywith the received light intensity. This voltage may be amplified by ahigh-impedance amplifier, preferably with low offset currents. Theexample of FIG. 18 a shows this amplification performed by a simple opamp with negative feedback via a voltage divider. The gain provided bythis arrangement can be readily recognized by one skilled in the art as1+(R_(f)/R_(g)). The op amp produces an isolated and amplified outputvoltage that increases, at least for a range, monotonically withincreasing light received at the light detection LED 1. Further in thisexample illustrative circuit, the output voltage of the op amp isdirected to LED100 via current-limiting resistor R100. The result isthat the brightness of light emitted by LED 100 varies with the level oflight received by LED 1.

For a simple lab demonstration of this rather remarkable fact, one maychoose a TL080 series (TL082, TL084, etc.) or equivalent opamp poweredby ±12 volts, 8100 of ˜1KΩ, and R_(f)/R_(g) in a ratio ranging from 1 to20 depending on the type of LED chosen. LED 100 will be dark when LED1is engulfed in darkness and will be brightly lit when LED1 is exposed tonatural levels of ambient room light. For best measurement studies, LED1could comprise a “water-clear” plastic housing (rather thancolor-tinted). It should also be noted that the LED1 connection to theamplifier input is of relatively quite high impedance and as such canreadily pick up AC fields, radio signals, etc. and is best realizedusing as physically small electrical surface area and length aspossible. In a robust system, electromagnetic shielding is advantageous.

The demonstration circuit of FIG. 18 a can be improved, modified, andadapted in various ways (for example, by adding voltage and/or currentoffsets, JFET preamplifiers, etc.), but as shown is sufficient to showthat a wide range of conventional LEDs can serve as pixel sensors for anambient-room light sensor array as may be used in a camera or otherroom-light imaging system. Additionally, LED 100 shows the role an LEDcan play as a pixel emitter of light.

FIG. 18 b shows a demonstration circuit for the photocurrent of the LED.For at least a range of light intensity levels the photocurrentgenerated by LED 1 increases monotonically with the received lightintensity. In this exemplary circuit the photocurrent is directed to anatively high-impedance op amp (for example, a FET input op amp such asthe relatively well-known LF 351) set up as an invertingcurrent-to-voltage converter. The magnitude of the transresistance(i.e., the current-to-voltage “gain”) of this invertingcurrent-to-voltage converter is set by the value of the feedbackresistor R_(f). The resultant circuit operates in a similar fashion tothat of FIG. 18 a in that the output voltage of the op amp increases, atleast for a range, monotonically with increasing light received at thelight detection LED. The inverting current-to-voltage converter invertsthe sign of the voltage, and such inversion in sign can be corrected bya later amplification stage, used directly, or is preferred. In othersituations it may be advantageous to not have the sign inversion, inwhich case the LED orientation in the circuit may be reversed, as shownin FIG. 18 c.

FIG. 18 d shows an illustrative demonstration arrangement in which anLED can be for a very short duration of time reverse biased and then ina subsequent interval of time the resultant accumulations of charge inthe junction capacitance of the LED are discharged. The decrease incharge during discharge through the resistor R results in a voltage thatcan be measured with respect to a predetermined voltage threshold, forexample as may be provided by a (non-hysteretic) comparator or(hysteretic) Schmitt-trigger. The resulting variation in discharge timevaries monotonically with the light received by the LED. Theillustrative demonstration arrangement provided in FIG. 18 d is furthershown in the context of connects to the bidirectional I/O pin circuitfor a conventional microprocessor. This permits the principal to bereadily demonstrated through a simple software program operating on sucha microprocessor. Additionally, as will be seen later, the very samecircuit arrangement can be used to variably control the emitted lightbrightness of the LED by modulating the temporal pulse-width of a binarysignal at one or both of the microprocessor pins.

FIG. 19 shows a simple illustrative variation of the circuit of FIG. 18a where a SPDT switch is introduced for controlling the function of LED1 at any given time. When the switch is flipped to the left, LED1 isilluminated at a brightness level determined by current limitingresistor R and applied voltage V+. During an interval of time when theswitch is flipped to the right position, LED1 serves as the light sensorfor the arrangement of FIG. 18 a. When LED 1 is emitting light, there isno measurement of light and the operation of the remainder of thecircuit, including that of LED 100, for that interval of time ismeaningless. As will be shown, however, this circuit can be reused forother LEDs and/or can be adapted to store the voltage associated with anearlier measurement so that LED 100 can continue to emit light of ahistorically-set brightness. The SPDT switch can be replaced with asimpler mode switching arrangements as will be seen.

FIG. 20 illustrates how two LEDs, here LED1 and LED2, can share theamplifier circuit and LED 100 while in alternating roles of emitting anddetecting light. In this illustrative circuit, a DPDT switch isconfigured as a reversing switch. When the switch is flipped to theleft, LED 1 serves as the photodetector for the arrangement of FIG. 18 awhile LED2 emits light at a brightness determined by current-limitingresistor R4 and the voltage applied to the “Emitted Light Intensity”input. When the switch is flipped to the right the roles of LED 1 andLED2 are exchanged. Should the voltage applied to the “Emitted LightIntensity” input be varied, the intensity of the light emitted bywhichever of LED 1 and LED2 is at that moment in emission mode (asdetermined by the position of the DPDT switch) can be responsivelycontrolled. Thus LED1 and LED2 can provide independently adjustablelevels of light intensity when in light emission mode.

FIG. 21 shows an illustrative variation of the arrangement of FIG. 20when a second (here a SPDT) switch is used to determine which of LED101and LED 102 is illuminated responsive to the light levels detected byLED1 or LED2. If the DPDT switch and the SPDT switch are operated in thesame direction simultaneously, LED 101 will light with an intensitydetermined by the light level received by LED1 and LED102 will lightwith an intensity determined by the level of light received by LED2respectively. When LED 1 or LED2 are not detecting light, they insteademit light with separately determined intensifies responsive to theirrespective “LED1 Intensity” and “LED2 Intensity” voltage inputs, andLED101 and LED 102 are respectively dark (as no current flows when theSPDT switch pole is not connected).

It is noted that in any of the previous examples, the SPDT and DPDTswitches may be mechanical (as suitable for a quick laboratorydemonstration) or may be implemented with analog semiconductor switchelements such as switching FETs, CMOS transistor configurations, etc.One skilled in the art will recognize that the switches described inFIGS. 18 a-21 may be implemented with components such as MPF102 JFETs,CD4016 and CD4066 analog switch integrated circuits, CD4051 multiplexerintegrated circuits, and/or other similar components. Thus, all thefunctions described thus far can be implemented entirely withsemiconductor elements. One skilled in the art will further recognizethat the opamps, resistors, and analog switches may all be fabricated ona common substrate with CMOS transistors that may be fabricated andinterconnected via standard photolithography electronic fabricationtechniques. Additionally, such circuits may be optimized for simplicity,function, and/or cost minimization.

FIG. 22 depicts a further illustrative variation of FIG. 21 wherein twoDPDT switches are operated together simultaneously (acting as a single4PDT switch). Based on the development thus far, it is clear to see thatwhen both switches are in the left position, LED101 and LED 102 emitlight of intensity responsive to the light received by LED1 and LED2,respectively. Further, it is equally clear to see that when the switchesare in the far right position LED 1 and LED2 emit light of intensityresponsive to the light received by LED 101 and LED 102, respectively.If the switches are electronic, they may be operated by a square waveoscillator (as shown in the figure). If the oscillator frequency exceeds˜25-30 Hz the light and dark modes of each LED will blend together asperceived by the human eye with each LED illuminated at approximatelyhalf the brightness as would be perceived if the switches were such thatthe LEDs were fully illuminated without interception. The 50% drop inbrightness resulting from each LED actively emitting light only 50% ofthe time. In this way LED1 and LED2 can reproduce a light distributionmeasured by LED101 and LED102 while LED101 and LED2 can reproduce alight distribution measured by LED1 and LED2, all of this humanlyperceived as occurring simultaneously.

As mentioned earlier, the illustrative SPDT, DPDT, and 4PDT modeswitching arrangements described thus far can be replaced with a simplerswitching configurations. In particular, in the detection circuit ofFIG. 18 a, the LED 1 connection to the amplifier input is of relativelyquite high impedance. Thus, a switched connection to a relatively muchlower impedance signal source will essentially fully dominate anycircuit effects of the LED 1 interconnection with the amplifier andallow LED 1 to be illuminated by the lower impedance signal source. Suchan analog-signal switch connection may be realized by a switching FETs,CMOS transistor configuration, etc., that permits operation of theswitch with a binary control signal.

FIG. 23 depicts an illustrative circuit arrangement wherein an LEDconnected to a detection buffer/amplifier (as in FIG. 18 a) is alsoconnected to a binary-signal-controlled analog switch element with high“off” impedance (such as a switching FETs, CMOS transistorconfiguration, etc.). When the binary control signal is of one state(for example, “0” or “low”) the analog switch provides a sufficientlyhigh impedance and does not appreciably load the LED voltage path to thedetection buffer/amplifier, and the output of the detectionbuffer/amplifier provides a signal response to the light intensityincident on the photo-active portion of the LED. When the binary controlsignal is of the other state (for example, “1” or “high”) the analogswitch provides a sufficiently low impedance path between the LED and alow-impedance emission signal source will essentially fully dominate anycircuit effects of the LED interconnection with the amplifier and allowthe LED to be illuminated responsive to the low impedance signal source.The low impedance signal source may be external and permit directconnection with the analog switch element, or may be a secondbuffer/amplifier which buffers and/or amplifies an incoming emissionlight intensity signal. During the time that the analog switch is closedso that the LED may emit light responsive to incoming emissionillumination signal, the output of the detection buffer/amplifier may beviewed as meaningless and ignored. In some embodiments, however, insteadof being ignored this signal may be used for other purposes (for examplediagnostics, feedback, etc.).

FIGS. 24 a-24 c depict exemplary state diagrams for the operation of theLED and the use of input signals and output signals described above.From the viewpoint of the binary mode control signal, the arrangement ofFIG. 23 has at only two states: a detection state and an emission state,as suggested in FIG. 24 a. From the viewpoint of the role of the LED ina larger system incorporating the circuit arrangement such as that ofFIG. 23, there may a detection state, an emission state, and an idlestate (where there is no emission nor detection occurring), obeyingstate transition maps such as depicted in FIG. 24 b or FIG. 24 c. At afurther level of detail, there are additional considerations:

-   -   To emit light, the binary mode control signal of the FIG. 23        example must be in “emit” mode (causing the analog switch to be        closed) and the emission light signal must be of sufficient        value to cause the LED to emit light (for example, so that the        voltage across the LED is above the “turn-on” voltage for that        LED).        -   If the binary mode control signal is in “emit” mode but the            emission light signal is not of such sufficient value, the            LED will not illuminate. This can be useful for brightness            control (via pulse-width modulation), black-screen display,            and other uses. In some embodiments, this may be used to            coordinate the light emission of neighboring LEDs in an            array while a particular LED in the array is in detection            mode.        -   If the emission light signal of such sufficient value but            the binary mode control signal is in “detect” mode, the LED            will not illuminate responsive to the emission light signal.            This allows the emission light signal to be varied during a            time interval when there is no light emitted, a property            useful for multiplexing arrangements.    -   During a time interval beginning with the change of state of the        binary mode control signal to some settling-time period        afterwards, the detection output and/or light emission level may        momentarily not be accurate.    -   To detect light, the binary mode control signal of the FIG. 23        example must be in “detect” mode (causing the analog switch to        be open). The detected light signal may be used by a subsequent        system or ignored. Intervals where the circuit is in detection        mode but the detection signal is ignored may be useful for        multiplexing arrangement, in providing guard-intervals for        settling time, to coordinate with the light emission of        neighboring LEDs in an array, etc.

FIG. 24 d depicts an exemplary state transition diagram reflecting theabove considerations. The top “Emit Mode” box and bottom “Detect Mode”box reflect the states of an LED from the viewpoint of the binary modecontrol signal as suggested by FIG. 24 a. The two “Idle” states (one ineach of the “Emit Mode” box and “Detect Mode” box) of FIG. 24 d reflect(at least in part) the “Idle” state suggested in FIG. 24 b and/or FIG.24 c. Within the “Emit Mode” box, transitions between “Emit” and “Idle”may be controlled by emit signal multiplexing arrangements, algorithmsfor coordinating the light emission of an LED in an array while aneighboring LED in the array is in detection mode, etc. Within the“Detect Mode” box, transitions between “Detect” and “Idle” may becontrolled by independent or coordinated multiplexing arrangements,algorithms for coordinating the light emission of an LED in an arraywhile a neighboring LED in the array is in detection mode, etc. Inmaking transitions between states in the boxes, the originating andtermination states may be chosen in a manner advantageous for details ofvarious multiplexing and feature embodiments. Transitions between thegroups of states within the two boxes correspond to the vast impedanceshift invoked by the switch opening and closing as driven by the binarymode control signal. In the Figure, the settling times between these twogroups of states are gathered and regarded as a transitional state.

As mentioned earlier, the amplitude of light emitted by an LED can bemodulated to lesser values by means of pulse-width modulation (PWM) of abinary waveform. For example, if the binary waveform oscillates betweenfully illuminated and non-illuminated values, the LED illuminationamplitude will be perceived roughly as 50% of the full-on illuminationlevel when the duty-cycle of the pulse is 50%, roughly as 75% of thefull-on illumination level when the duty-cycle of the pulse is 75%,roughly as 10% of the full-on illumination level when the duty-cycle ofthe pulse is 10%, etc. Clearly the larger fraction of time the LED isilluminated (i.e., the larger the duty-cycle), the brighter theperceived light observed emitted from the LED. FIG. 25 illustrates onemethod of generating a pulse-width modulated waveform by using acomparator to compare the values of an analog voltage input varying overa range of voltages that is also the same as the range of voltagesprovided by a ramp waveform oscillator. Another method of generating apulse-width modulated waveform is to use one or more timers operating inor operated by a microprocessor program.

In the demonstrative arrangement depicted in FIG. 22, the duty cycle wasset at 50% (sans negligible switching times to operate the switch) sothat each LED is a detector for half the time and an emitter for theother have of the time. However, detections can be made quickly, sospending so much of the duty-cycle detecting prevents an LED in the FIG.22 arrangement from being a very efficient light emitter. Thus theillustrative system would behave better LED displays on one side of thearrangement of FIG. 22 and worse LED displays on the other side of thearrangement if the duty-cycle was changed from 50% to a number smalleror larger value (a smaller duty-cycle value brightening the LEDsdisplaying on one side of FIG. 22 and dimming the LEDs displaying on theother side of FIG. 22, and a large duty-cycle value resulting in anexchange of roles). FIG. 26 a illustrates a mode control signalpossessing a non-symmetric duty-cycle that favors emission time durationand limits detection time duration, as may be applied to the arrangementof FIG. 23, represented in a simplified form by FIG. 26 b. Thebuffers/amplifiers depicted may be voltage or current. In somecurrent-drive arrangement a current-limiting resistor is not needed,while in voltage-drive arrangements a current-limiting resistor may berequired.

FIG. 27 a depicts the exemplary use of a sample/hold arrangement whereinthe analog switch briefly closes to direct the analog emission controlvoltage to the capacitor C. The capacitor C charges to match thisvoltage and retains this voltage value after the switch is opened. Aunity gain amplifier reproduces this voltage without loading (and thuswithout discharging) the charge stored in the capacitor. Using thisapproach, a very short measurement time-interval may be used fordetection, long enough to allow a shorter-duration sample/hold operationto be made and complete, and the resulting voltage may be held for aconsiderably longer emission time-interval. Exemplary relative timingfor the detection signal and the sampling control signal as justdescribed is depicted in FIG. 27 b.

FIG. 28 shows the combination of the arrangement of FIG. 27 a with thearrangement of FIG. 26 b. Switch SWa provides emit/detect mode controlwhile SWb provides sampling control. The active elements of such acircuit can be fabricated from two switching FETs and two amplifyingMOSFETs, with four MOSFETs, or in other ways.

Further, the arrangement of FIG. 27 a may be expanded so as to providean exemplary analog demultiplexing function as shown in FIG. 29. Here, aplurality of normally-open analog switches are sequentially closed forbrief disjointed intervals of time. The associated plurality ofcapacitors retain the sampled voltages until the next associated switchclosing, each of which is respectively buffered by an associated bufferamplifier which drives an associated illuminating LED at a brightnessassociated with the retained sampled voltage value.

Additionally, an arrangement such as that depicted in FIG. 30 can beused to sequentially select from a plurality of detection amplifieroutputs. Here again, a plurality of normally-open analog switches aresequentially closed for brief disjointed intervals of time. Thearrangement of FIGS. 29 and 30 can be combined, as shown in FIG. 31, tomultiplex a plurality of detection signals onto a common analog signalpath which is then demultiplexed into a plurality of sample/holdcircuits. This arrangement can be synchronized with the mode controlsignal so that each LED very briefly provides periodically updateddetection measurement and is free to emit light the rest of the time.

Moreover, the interconnected plurality switch arrangement of FIG. 30 canbe used to provide a selectable grounding capability for atwo-dimensional array of LEDs as depicted in FIG. 32 a. Here again, theplurality of normally-open analog switches are sequentially closed forbrief disjointed intervals of time. This allows the selection of aparticular subset (here, a column) of LEDs to be grounded while leavingall other LEDs in the array not connected to ground. Each of thehorizontal lines then can be used to connect to exactly one grounded LEDat a time. The plurality of normally-open analog switches in FIG. 32 amay be controlled by an address decoder so that the selected subset canbe associated with a unique binary address, as suggested in FIG. 32 b.The combination of the plurality of normally-open analog switchestogether with the address decoder form an analog line selector. Byconnecting the line decoder's address decoder input to a counter, thecolumns of the LED array can be sequentially scanned.

Based on these constructions, arrangements, and principles, FIG. 33depicts an exemplary combination of the arrangements of FIG. 28, FIG.30, and FIG. 32 b together to form a highly scalable LED array displaythat also functions as a light field detector. The various multiplexingswitches in this arrangement can be synchronized with the line selectorand mode control signal so that each LED very briefly providesperiodically updated detection measurement and is free to emit light therest of the time.

The arrangement of FIG. 33 can be reorganized so that the LED, modecontrol switch, capacitor, and amplifiers are collocated, for example asin the illustrative exemplary arrangement of FIG. 34 a. Such anarrangement can be implemented with, for example, three MOSFET switchingtransistor configurations, two MOSFET amplifying transistorconfigurations, a small-area/small-volume capacitor, and an LED element(that is, five transistors, a small capacitor, and an LED). This can betreated as a cell which is interconnected to multiplexing switches andcontrol logic.

The arrangement of FIG. 33 can be reorganized to decentralize themultiplexing structures so that the LED, mode control switch,multiplexing and sample/hold switches, capacitor, and amplifiers arecollocated, for example as in the illustrative exemplary arrangement ofFIG. 34 b. Such an arrangement can be implemented with, for example,three MOSFET switching transistor configurations, two MOSFET amplifyingtransistor configurations, a small-area/small-volume capacitor, and anLED element (that is, five transistors, a small capacitor, and an LED).This can be treated as a cell whose analog signals are directlyinterconnected to busses. Other arrangements are also possible.

The discussion and development thus far are based on the analog circuitmeasurement and display arrangement of FIG. 18 a that in turn leveragesthe photovoltaic properties of LEDs. With minor modifications clear toone skilled in the art, the discussion and development thus far can bemodified to operate based on the analog circuit measurement and displayarrangement of FIGS. 18 b/18 c that in turn leverage the photocurrrentproperties of LEDs. FIG. 35 a, FIG. 35 b and FIG. 35 c provide guidanceon how the digital circuit measurement and display arrangement of FIG.18 d (that in turn leverages discharge times for accumulations ofphoto-induced charge in the junction capacitance of the LED) can beadapted into the construction developed thus far. FIG. 35 a adapts FIG.18 d to additional include provisions for illuminating the LED with apulse-modulated emission signal. Noting that the detection processdescribed earlier in conjunction with FIG. 18 d can be confined tounperceivably short intervals of time, FIG. 35 b illustrates how apulse-width modulated binary signal may be generated during LEDillumination intervals to vary LED emitted light brightness. FIG. 35 cillustrates an adaptation of the tri-state andSchmitt-trigger/comparator logic akin to that illustrated in themicroprocessor I/O pin interface that may be used to sequentially accesssubsets of LEDs in an LED array as described in conjunction with FIG. 32a and FIG. 32 b.

A plurality of cells such as the exemplary arrangements of FIG. 34 a, 34b, individual LEDs in matrices interfaced directly to logic circuit ascompatible with the arrangement of FIG. 35 a, and other possiblearrangements can be incorporated into the larger system arranges such asthose depicted in FIG. 36 a and FIG. 36 b. Both of these figures depictan exemplary arrangement that takes the light field measurement andpresents it to image formation algorithms, for example as based on thesystems, methods, and techniques described earlier and produces imagedata out. The image data may serve as motion video or may comprise astatic image sampling a scene at an earlier point in time. FIG. 36 adepicts a two-way LED array system that serves as both a camera and adisplay. FIG. 36 b depicts a simplified one-way LED array system thatserves as a camera.

FIG. 37 a and FIG. 37 b depict a view of the resulting synergisticcombinations of methods and elements as provided for by the invention.The signal processing architecture 2950 comprises image formationalgorithms and associated DSP 2940 as well as aspects of the vignettemicro-optical arrangements 2930 of the sensor or integratedsensor/display 2920. The manufacturing architecture 2960 comprisessensor or integrated sensor/display 2920 as well as associated vignettemicro-optical arrangements 2930 and DSP, power supply, shielding,enclosures, etc. For a two-way system that serves as both a camera and adisplay, the integrated sensor/display 2920 may comprise an LED array asan emitter/detector 2901 or another type of emitter/detector array 2902as illustrated in FIG. 37 a. For a one-way system that as a camerawithout a common bulk lens, the sensor 2920 may comprise an LED array asa light field detector 2903 or another type of light field detectorarray 2904 as illustrated in FIG. 37 b.

As indicated earlier, Forest Mims reported in 1992 the light spectralselectivity of LED photovoltaic generation very nearly matches theemitted light spectrum of the LED. It is also known that light-inducedLED photocurrents (which are also the process charging the junctioncapacitance in the arrangement of FIG. 18 d) also exhibit light spectralselectivity of that very nearly matches the emitted light spectrum ofthe LED. This suggests use of color LEDs as color-sensitive sensors.Thus it would be opportune and advantageous for a color LED array suchas currently emerging for LED televisions to also serve as a color lightfield detector for use with the synthetic image formation systems andmethods described earlier.

FIG. 38 a illustrated the human eye sensitivity to the spectrum ofwavelengths of visible light. The human eye comprises “cone” and “rod”constituent densely-packed spatially-distributed structures, wherein therods detect incoming light brightness (luminance) while three types ofcones discern three primary colors. In contrast, the spectral bandwidthof a traditional LED is typically 50 nm. Positioning three approximately50 nm basspands in the portions of the visible spectrum near peaks ofthe cone responses yet away from areas of significant spectral overlapwould result in a spectral placement such as that depicted in theexample of FIG. 38 b. For use as a sensor, clearly there are gaps in thespectral coverage; this will be revisited shortly. For use as a monitor,one approach to a good performance would be the arrangement depicted inFIG. 38 c. Here an “ideal sensor” mimics the cone responses of the humaneye and provides resultant red, green, and blue measurements. Thesemeasurements in turn would drive the amplitudes of the narrowband LEDemissions from FIG. 38 b, which in turn stimulate the three coneresponses in areas of relatively low sensitivity overlap.

Thus as a color light field sensor the wider spectral characteristics ofFIG. 38 a are desirable, while for a display the narrower spectralcharacteristics of FIG. 38 a are desirable. It is known that through useof various techniques the emission spectrum of an LED can be widened.For example, it would be possible to obtain the response curves of FIG.39 a as a mimic of the cone responses of the human eye depicted in FIG.38 a. As a monitor, however emission spectral characteristics such asthat of FIG. 38 b are desirable. It has long been known (Yariv, OpticalElectronics, Holt Rinehart, and Winston, 1971) that photocurrentscomprise at least three independent photoresponsive effects thatcontribute to light-stimulated current flow in a PN diode junction. Thusit may be possible to adjust each color LED used in an array so that thedetection spectrum of each comprises wider passbands, as in FIG. 39 a inapproximating the curves of FIG. 38 a, while the emission spectra ofeach comprises narrower passbands, as in FIG. 39 b in approximating thecurves of FIG. 38 b. The invention therefore provides for an embodimentin which each color LED used in an array so that the detection spectrumof each comprises wider passbands while the emission spectra of eachcomprises narrower passbands.

Should it not be possible, practical, or cost effective to obtain suchLEDs, another approach is depicted in FIG. 40 a and FIG. 40 b. Here, thenarrowband detection spectral LED properties imposed by desirableemission characteristics such as in FIG. 38 b or FIG. 39 b can besupplemented by additional LEDs providing additional coverage bands. Anexample of this is depicted in FIG. 40 a wherein the red, green, andblue sensitivities inherited from the FIG. 39 b emission spectra ofemission LEDs are supplemented with additional LEDs with sensitivitybands W, X, Y, and Z. In one embodiment, the invention provides fordetection outputs of these additional supplemental LEDs to be combinedwith the detection outputs of the three emission LEDs as suggested byFIG. 40 b. In one variation of the embodiment, all the LEDs aretransparent and stacked vertically. In one variation of the embodiment,all the LEDs are immediately adjacent on the plane of the array. In onevariation of the embodiment, the electrical connections of thesupplemental LEDs are switched by circuitry within the cells exemplarilydepicted in FIGS. 34 a-34 b and FIGS. 36 a-36 b so as to connect to theemission LEDs only when the emission LEDs are operating in detectionmode. In one variation of the embodiment, the detection measurementsmade by the supplemental LEDs are combined numerically with thedetection measurements made by the emission LEDs.

Further, the LED detection capabilities of emission LEDs may not have adesirable amplitude response with respect to received light amplitude orintensity. In this case, the invention provides for the application oftraditional or more advanced forms of nonlinear “gamma correction” tothe detected signals provided by the LED array. The gamma correctionneed not be limited to the traditional family of binomial power curvesbut instead be any appropriate nonlinearity that proves useful. The“gamma correction” is applied to measurement data streams provided bythe detection multiplexing arrangement (rather than separatelyimplemented for each LED). In some embodiments the “gamma correction” ismade advantageously made separately for each color detection signal assuggested by FIG. 41 a. In other embodiments the “gamma correction” ismade advantageously made on the vector of color detection signals assuggested by FIG. 41 b, allowing for cross-coupled corrections. Thesecross-coupled corrections can be useful in color handling issues inregions of the spectrum where there is considerable overlap in the humaneyes cone response curves.

In contemporary optoelectronic device design and manufacturing, thechoice of optical diode materials, structures, and fabricationtechniques are separately optimized to obtain markedly different resultsfor LEDs and photodiodes. This is illustrated in FIG. 42, whereinoptical diode materials, structures, and fabrication techniques 4223 areregarded as “variables” resulting in devices having a range ofattributes that are simplified down to emission performance 4201,detection performance 4202, and cost 4203 in a metric space of devicerealizations 4200. The figure illustrates broadly isolated planarregions for LEDs 4211 and photodiodes 4212 in this metric space ofdevice realizations, each region spanning ranges of cost andperformance. The invention provides for co-optimizations in bothemission performance and detection performance against cost, representedby the volume region 4213. Such co-optimizations in both emissionperformance and detection may comprise adjustments and divergence ofemission and detection spectral characteristics, co-locationconsiderations, switching transistor interfacing, etc.

The invention provides for individual LEDs to be arranged in a varietytwo-dimensional arrays of a variety of planar and non-planar manners onrigid or flexible substrates. Individual LEDs or stacks of LEDs may beshared and distributed in a variety of patterns, for example as a simplelattice of circular devices as suggested in the exemplary arrangement ofFIG. 43 a, a simple lattice of square or rectangular devices as shown inFIG. 43 b, the more complex lattice of hexagonal devices as shown inFIG. 43 c, etc. Each device comprises LEDs or other light sensor/emitterelements and, in many embodiments, associated cell electronicsexemplarily depicted in FIGS. 34 a-34 b and FIGS. 36 a-36 b. The layoutarrangements are coordinated with micro-vignette and other micro-opticsfeatures described earlier.

The invention provides for synthetically varying the aperture size inthe image formation algorithms. This is useful for providing syntheticimaging with different depths of field. For example, as seen in FIGS. 9a-9 d, more distant scene elements require more sensor resolution todiscern the incoming light field than near scene elements. In anexemplary embodiment, far-field scene element synthetic image formationuse individual sensors elements (such as 3601), A somewhat closer sceneelement may use a linear combination of measurements from fourindividual sensors elements (such as suggested by 3602) in syntheticimage formation. Similarly, a yet closer scene element may use a linearcombination of measurements from nine individual sensors elements (suchas suggested by 3603) in synthetic image formation, a yet closer sceneelement may use a linear combination of measurements from sixteenindividual sensors elements (such as suggested by 3604) in syntheticimage formation, etc. It is noted that such linear combinations ofmeasurements from a plurality of individual sensors elements can beimplemented in such a way to permit overlap, as suggested by the overlapof the envelopes 3605 and 3606.

An LED in the array, when in light detecting mode, will respond toincoming light from any variety of sources, including reflected lightthat provided by neighboring emitting-mode LEDs.

The invention also provides for adaptations of the arrangement describedthus far so as to permit illumination from neighboring illuminating-modeLEDs to be omitted or attenuated. This may be accomplished by modulatingthe illumination signals in some fashion (for example amplitudemodulation via a distinctive signal with high enough frequency contentso as to be not visually perceptual, phase or frequency modulation of anamplitude modulated signal, etc.) A light-detection signal obtained bythe LEDs operating in light detection mode can be analyzed in real-timefor the presence of this carrier and the light-detection signal can thenhave this corresponding component removed algebraically, by means ofvarious filters, or other methods. In some embodiments, the modulationtechnique may be chosen so that only simple filtering is required toremove this component with no need for real-time signal analysis. Thiscan result in removal of the effects of illumination by the screen, forexample if the screen image comprises colors or patterns the reflectionsfrom the screen can be attenuated or removed. In yet other embodiments,multiplexing of image detection and image emission may be arranged sothe scene viewed is not illuminated by light emitted from another partof the LED display.

In another modality, the invention also provides for adaptations of thearrangement described thus far so to permit neighboringilluminating-mode LEDs to be used or adapted to provide somewhatcolor-corrected near-field lighting. In one embodiment this is done byproviding a known-base illumination level of all emitted colors toproduce uniform white light for a brief time interval and perform alldetection operations only during this white light emission interval. Inanother embodiment this is done by modulating the intensity of a knownadded base level of continuously emitted white light with a modulationsignal that can be recognized and filtered from the measurementsdirected to image formation, said modulation, detection, and filteringas described in the preceding paragraph.

Although the present invention has been described in connection withparticular preferred embodiments and examples, it is to be understoodthat many modifications and variations can be made in hardware,software, operation, uses, protocols and data formats without departingfrom the scope to which the inventions disclosed herein are entitled.Accordingly, the present invention is to be considered as including allapparatus and methods encompassed by the appended claims.

1. A system for numerical image formation and for refinement fromoverlapping measurement vignettes captured by an image sensor array withassociated micro-optical arrangements, the system comprising: an imagesensor comprising the plurality of micro-optical arrangements thatcreates a plurality of light detection signals in response to aplurality of optically vignetted portions of an incoming light fieldassociated with an observed scene, wherein each of the plurality ofoptically vignetted portions is produced by individual ones of themicro-optical arrangements; a multiplexing arrangement for gatheringsaid plurality of light detection signals into a stream of measurementvalues responsive to an incoming light field associated with an observedscene; a processor for receiving the stream of measurement values, theprocessor executing a numerical image formation algorithm comprisinglinear mathematical transformations on the stream of measurement valuesto produce at least one synthesized digital image; and an output fortransferring said at least one synthesized digital image to an externalsystem; wherein the processor produces a visual image responsive to theincoming light field associated with the observed scene.
 2. The systemof claim 1 wherein the light detection signals are produced by circuitsmeasuring light-induced electrical behavior of LEDs.
 3. The system ofclaim 2 wherein the light-induced electrical behavior is a photocurrent.4. The system of claim 2 wherein the light-induced electrical behavioris a light-induced voltage.
 5. The system of claim 2 wherein thelight-induced electrical behavior is a light-induced charge.
 6. Thesystem of claim 2 wherein the light-emitting diode light sensorsadditionally configured to emit light act as a visual image display. 7.The optical-electronic array of claim 6 wherein the multiplexingarrangement is further configured to multiplex individualoptical-electronic elements in at least a display mode and a sensormode.
 8. The system of claim 1 wherein the light detection signals arefurther processed by a nonlinear “gamma correction” element.
 9. Thesystem of claim 1 wherein the two-dimensional array of light sensorscomprises a color LED array as a color light-field sensor.
 10. Thesystem of claim 7 wherein the light detection signals are furtherprocessed by a nonlinear “gamma correction” element applied separatelyfor each color detection signal.
 11. The system of claim 7 wherein thelight detection signals are further processed by a nonlinear “gammacorrection” element applied to the plurality of color detection signals,the treated as a plurality of color detection signals vector of colordetection signals.
 12. The system of claim 1 wherein the linearmathematical transformations employ linear combinations of measurementsfrom a plurality of measurement values.
 13. The system of claim 12wherein the envelopes of at least two apertures overlap.
 14. The systemof claim 1 wherein the numerical image formation algorithm isadjustable, the adjustment responsive to configuration commands.
 15. Thesystem of claim 1 wherein the numerical image formation algorithm isadjustable, the adjustment responsive to control signals.
 16. The systemof claim 1 additionally comprising subsequent image processingoperations.
 17. The system of claim 16 wherein the subsequent imageprocessing operations are adjustable, the adjustment responsive toconfiguration commands.
 18. The system of claim 16 wherein thesubsequent image processing operations are adjustable, the adjustmentresponsive to control signals.